Serving ML Predictions in batch and real-time

2 hours Free

Overview

Duration is 1 min

In this lab, you run Dataflow pipelines to serve predictions for batch requests as well as streaming in real-time.

What you learn

In this lab, you write code to:

Create a prediction service that calls your trained model deployed in Cloud to serve predictions
Run a Dataflow job to have the prediction service read in batches from a CSV file and serve predictions
Run a streaming Dataflow pipeline to read requests real-time from Cloud Pub/Sub and write predictions into a BigQuery table

Setup

For each lab, you get a new Google Cloud project and set of resources for a fixed time at no cost.

Make sure you signed into Qwiklabs using an incognito window.
Note the lab's access time (for example, and make sure you can finish in that time block.

When ready, click .
Note your lab credentials. You will use them to sign in to the Google Cloud Console.
Click Open Google Console.
Click Use another account and copy/paste credentials for this lab into the prompts.

Accept the terms and skip the recovery resource page.

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

In the Cloud Console, in the top right toolbar, click the Activate Cloud Shell button.

Cloud Shell icon

Click Continue.

It takes a few moments to provision and connect to the environment. When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. For example:

Cloud Shell Terminal

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

You can list the active account name with this command:

gcloud auth list

(Output)

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)

(Example output)

Credentialed accounts:
 - google1623327_student@qwiklabs.net

You can list the project ID with this command:

gcloud config list project

(Output)

[core]
project = <project_ID>

(Example output)

[core]
project = qwiklabs-gcp-44776a13dea667a6

For full documentation of gcloud see the gcloud command-line tool overview.

Check project permissions

Before you begin your work on Google Cloud, you need to ensure that your project has the correct permissions within Identity and Access Management (IAM).

In the Google Cloud console, on the Navigation menu (), click IAM & Admin > IAM.
Confirm that the default compute Service Account {project-number}-compute@developer.gserviceaccount.com is present and has the editor role assigned. The account prefix is the project number, which you can find on Navigation menu > Home.

If the account is not present in IAM or does not have the editor role, follow the steps below to assign the required role.

In the Google Cloud console, on the Navigation menu, click Home.
Copy the project number (e.g. 729328892908).
On the Navigation menu, click IAM & Admin > IAM.
At the top of the IAM page, click Add.
For New members, type:

{project-number}-compute@developer.gserviceaccount.com

Replace {project-number} with your project number.

For Role, select Project (or Basic) > Editor. Click Save.

Creating the virtual environment

Execute the following command to download and update the packages list.

sudo apt-get update

Python virtual environments are used to isolate package installation from the system.

sudo apt-get install virtualenv

If prompted [Y/n], press Y and then Enter.

virtualenv -p python3 venv

Activate the virtual environment.

source venv/bin/activate

Copy trained model

Step 1

Set necessary variables and create a bucket:

REGION=us-central1
BUCKET=$(gcloud config get-value project)
TFVERSION=2.1
gsutil mb -l ${REGION} gs://${BUCKET}

Step 2

Copy trained model into your bucket:

gsutil -m cp -R gs://cloud-training-demos/babyweight/trained_model gs://${BUCKET}/babyweight

Deploy trained model

Step 1

Set necessary variables:

MODEL_NAME=babyweight
MODEL_VERSION=ml_on_gcp
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/babyweight/export/exporter/ | tail -1)

Step 2

Set the region to global by executing the following command:

gcloud config set ai_platform/region global

Step 3

Deploy trained model:

gcloud ai-platform models create ${MODEL_NAME} --regions $REGION
gcloud ai-platform versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version $TFVERSION

Browse lab files

Duration is 5 min

Step 1

Clone the course repository:

cd ~
git clone https://github.com/GoogleCloudPlatform/training-data-analyst

Step 2

In Cloud Shell, navigate to the folder containing the code for this lab:

cd ~/training-data-analyst/courses/machine_learning/deepdive/06_structured/labs/serving

Step 3

Run the what_to_fix.sh script to see a list of items you need to add/modify to existing code to run your app:

./what_to_fix.sh

As a result of this, you will see a list of filenames and lines within those files marked with TODO. These are the lines where you have to add/modify code. For this lab, you will focus on #TODO items for .java files only, namely BabyweightMLService.java : which is your prediction service.

How the code is organized

Prediction service

In this section, you fix the code in BabyweightMLService.java and test it with the run_once.sh script that is provided. If you need help with the code, look at the next section that provides hints on how to fix code in BabyweightMLService.java.

Step 1

You may use the Cloud Shell code editor to view and edit the contents of these files.

In Cloud Shell, click the Open Editor icon on the top right.

Step 2

After it is launched, navigate to the following directory:

training-data-analyst/courses/machine_learning/deepdive/06_structured/labs/serving/pipeline/src/main/java/com/google/cloud/training/mlongcp

Step 3

Open the BabyweightMLService.java files and replace #TODOs in the code.

Step 4

Once completed, go into your Cloud Shell and run the run_once.sh script to test your ML service.

cd ~/training-data-analyst/courses/machine_learning/deepdive/06_structured/labs/serving
./run_once.sh

Serve predictions for batch requests

This section of the lab calls AddPrediction.java that takes a batch input (one big CSV), calls the prediction service to generate baby weight predictions and writes them into local files (multiple CSVs).

Step 1

In your Cloud Shell code editor, open the AddPrediction.java file available in the following directory:

training-data-analyst/courses/machine_learning/deepdive/06_structured/labs/serving/pipeline/src/main/java/com/google/cloud/training/mlongcp

Step 2

Look through the code and notice how, based on input argument, it decides to set up a batch or streaming pipeline, and creates the appropriate TextInputOutput or PubSubBigQuery io object respectively to handle the reading and writing.

Step 3

Test batch mode by running the run_ontext.sh script provided in the lab directory:

cd ~/training-data-analyst/courses/machine_learning/deepdive/06_structured/labs/serving

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

./run_ontext.sh

Serve predictions real-time with a streaming pipeline

In this section of the lab, you will launch a streaming pipeline with Dataflow, which will accept incoming information from Cloud Pub/Sub, use the info to call the prediction service to get baby weight predictions, and finally write that info into a BigQuery table.

Step 1

On your GCP Console's left-side menu, go into Pub/Sub and click the CREATE TOPIC button on top. Create a topic called babies.

Step 2

Back in your Cloud Shell, modify the script run_dataflow.sh to get Project ID (using --project) from command line arguments, and then run as follows:

cd ~/training-data-analyst/courses/machine_learning/deepdive/06_structured/labs/serving
./run_dataflow.sh

This will create a streaming Dataflow pipeline.

Step 3

Back in your GCP Console, use the left-side menu to go into Dataflow and verify that the streaming job is created.

Step 4

Next, click on the job name to view the pipeline graph. Click on the pipeline steps (boxes) and look at the run details (like system lag, elements added, etc.) of that step on the right side.

This means that your pipeline is running and waiting for input. Let's provide input through the Pub/Sub topic.

Step 5

Copy some lines from your example.csv.gz:

cd ~/training-data-analyst/courses/machine_learning/deepdive/06_structured/labs/serving
zcat exampledata.csv.gz

Step 6

On your GCP Console, go back into Pub/Sub, click on the babies topic, and then click on Publish message button on the top. In the message box, paste the lines you just copied from exampledata.csv.gz and click on Publish button.

Step 7

You may go back into Dataflow jobs on your GCP Console, click on your job and see how the run details have changed for the steps, for example click on write_toBQ and look at Elements added.

Step 8

Lets verify that the predicted weights have been recorded into the BigQuery table.

Open BigQuery Console

In the Google Cloud Console, select Navigation menu > BigQuery:

The Welcome to BigQuery in the Cloud Console message box opens. This message box provides a link to the quickstart guide and lists UI updates.

Click Done.

The BigQuery console opens.

Look at the Explorer tab and expand your GCP Project ID by clicking the down arrow next to it. You should see the babyweight dataset. Click on the down arrow to its left, and you should see your predictions table.

Step 9

Type the query in the query EDITOR to retrieve rows from your predictions table.

SELECT * FROM babyweight.predictions LIMIT 1000

Step 10

Click the Run button. Notice the predicted_weights_pounds column in the result.

Step 11

Remember that your pipeline is still running. You can publish additional messages from your example.csv.gz and verify new rows added to your predictions table. Once you are satisfied, you may stop the Dataflow pipeline by going into your Dataflow Jobs page, and click the Stop button on the top. Select Drain and click Stop Job.

End your lab

When you have completed your lab, click End Lab. Qwiklabs removes the resources you’ve used and cleans the account for you.

You will be given an opportunity to rate the lab experience. Select the applicable number of stars, type a comment, and then click Submit.

The number of stars indicates the following:

1 star = Very dissatisfied
2 stars = Dissatisfied
3 stars = Neutral
4 stars = Satisfied
5 stars = Very satisfied

You can close the dialog box if you don't want to provide feedback.

For feedback, suggestions, or corrections, please use the Support tab.

©2020 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.

Overview Setup Creating the virtual environment Copy trained model Deploy trained model Browse lab files How the code is organized Prediction service Serve predictions for batch requests Serve predictions real-time with a streaming pipeline End your lab

Welcome to Your First Lab!

Skip this video

Serving ML Predictions in batch and real-time

Sandhya Aparna Pashikanti

Serving ML Predictions in batch and real-time

Overview

What you learn

Setup

Activate Cloud Shell

Check project permissions

Creating the virtual environment

Copy trained model

Step 1

Step 2

Deploy trained model

Step 1

Step 2

Step 3

Browse lab files

Step 1

Step 2

Step 3

How the code is organized

Prediction service

Step 1

Step 2

Step 3

Step 4

Serve predictions for batch requests

Step 1

Step 2

Step 3

Serve predictions real-time with a streaming pipeline

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

Step 7

Step 8

Open BigQuery Console

Step 9

Step 10

Step 11

End your lab

Welcome to Your First Lab!